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Abstract 



We have searched for the two-body decay of the B meson to a hght pseu- 
doscalar meson h = tt^ , , and a massless neutral weakly-interacting 
particle such as the familon, the Nambu-Goldstone boson associated with 
a spontaneously broken global family symmetry. We find no significant sig- 
nal by analyzing a data sample containing 9.7 million BB mesons collected 
with the CLEO detector at the Cornell Electron Storage Ring, and set a 90% 
C.L. upper limit of 4.9 x 10"^ and 5.3 x 10"^ on the branching fraction for 
the decays B^ h^X^ and B^ — > K^X^ , respectively. These upper limits 
correspond to a lower bound of ~ 10*^ GeV on the family symmetry breaking 
scale involving the third generation of quarks. 
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The origin of family replication remains one of the major puzzles in particle physics. 
Why do we have three families of fermions, which are indistinguishable with respect to 
the strong and electroweak interactions? Neither the Standard Model (even incorporating 
the Higgs mechanism) nor its extension by various unification schemes in the framework 
of one family (SU(5), SO (10)) is able to provide a deep physical reason for the existence 
of the mass hierarchy among the generations and the weak mixing of quarks and leptons. 
In the absence of a concrete model, it is natural to assume that the underlying theory 
possesses a "horizontal" family symmetry which is spontaneously broken at some large energy 
scale. Among several possibilities, the most attractive is the assumption of a global (and 
continuous) flavor symmetry [|^. This symmetry, under some conditions 0], automatically 
induces the Peccei-Quinn sjTiimetry [Q, and thus provides a solution for the strong CP 
problem. The spontaneous symmetry breaking of a continuous and global family symmetry 
implies the existence of neutral massless Nambu-Goldstone bosons , called familons, which 
can have flavor- conserving as well as flavor- changing couplings with the fermions [|T],0|. 

Flavor-changing couplings between the familon and fermions induce decays, such as 
— s> 7r+X° or — > e~*"(7)X°, that have been studied experimentally 0,0]. Upper lim- 
its on the rate of these decays led to the lower bounds on the family symmetry breaking 
scale involving the first two generations: ~ 10^^ GeV and ~ 10^ GeV in the hadronic and 
the leptonic sector, respectively. In contrast, bounds on the flavor scale involving the third 
generation are less thoroughly studied experimentally, although, some theoretical models 
suggest that the familon couples preferentially to the third generation 0. The upper limits 
for r — > iX^ {£ = e, fi) |^ led to a lower bound on the family symmetry breaking scale 
F ^ 10^ GeV in the leptonic sector, and no bounds have been reported in the hadronic 
sector. 

Familon couplings to the third generation are also of interest from a cosmological point of 
view. A massive unstable neutrino (typically the tau-neutrino) was proposed to decay into 
a lighter neutrino and a massless boson, such as a familon, in several cosmological scenarios 
related to big-bang nucleosynthesis |jlO|, and large scale structure formation in order to 



obtain a reasonable agreement between theory and observation. Since the process is 
related to the decay modes r — > £/ and b — > qdf {qd = d, s) through S'U{2)l and SU(5) GUT 
gauge symmetries, searches for the latter decay modes can test the cosmological scenarios 
as well §. 

The decay of the b quark b -—>■ q^f would lead to the decay B —>■ hf {h = n, K) through 
vector coupling and B V f (V = p, K*) through axial coupling, respectively. The purpose 
of this study is to search for the B^ — ^ h^X^ and B^ — >■ K^X^ decays, where X^ is any 
neutral massless weakly- interacting particle including the familon, using the CLEO data set. 
The lack of a signal allows us to obtain a constraint on the vector coupling of the familon 
to third generation hadrons for the first time. (The analysis is sensitive to new physics 
including massless weakly-interacting neutral particles as well.) The partial width F of the 
decay is related to the family symmetry breaking scale F through the formula 



Fi(0) 



where Mb-, Tn^ are the masses of the mesons involved in the decay process, gy is the vector 
type coupling constant, Tm{s) are the generators of the broken symmetry, and -Fi(O) is the 
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form factor 0. 

The data analyzed in this study were collected with the CLEO detector at the Cor- 
nell Electron Storage Ring (CESR), a symmetric e~^e~ collider. The components of the 



detector |12| most relevant to this analysis are the charged particle tracking system, the 



Csl electromagnetic calorimeter, and the muon detector. Trajectories of charged particles 
were reconstructed using a system of three concentric wire chambers (a 6-layer straw tube 
chamber, a 10-layer precision drift chamber, and a 51-layer main drift chamber) covering 
95% of the total solid angle, operating in an axial solenoidal magnetic field of 1.5 T. The 
main drift chamber also provided a measurement of the specific ionization loss [dE/dx) used 
for particle identification. Photons were detected by a CsI(Tl) electromagnetic calorimeter 
covering 98% of 4tt. The muon chambers consisted of proportional counters embedded at 
various depths in the steel absorber. Approximately 2/3 of the data were collected with an 
upgraded detector, in which the innermost straw tube chamber was replaced with a three- 
layer, double-sided silicon vertex detector |TB|, and the gas in the main drift chamber was 



changed from an argon-ethane to a helium-propane mixture. These modifications led to an 
improved particle identification and momentum resolution. 

The results in this Letter are based upon an integrated luminosity of 9.2 fb~^ of e~^e~ 
data corresponding to 9.7 million BB meson pairs collected at the T(4S) resonance energy 
of 10.58 GeV ( "on-resonance sample") and 4.6 fb~^ at 60 MeV below the T(4S) resonance 
( "off- resonance sample"). The study of the off-resonance sample enables us to statistically 
subtract the continuum (e"''e^ ^ qq, q = u, d, c, s) background contribution from the on- 
resonance sample. In order to study signal reconstruction efficiency and to optimize selection 
criteria we generated Monte Carlo simulated samples with a GEANT-based Jl^ simulation 
of the CLEO detector response. Simulated data samples were processed in a similar manner 
as the data. 

The experimental signature of the the B^ — > h^X^ and B^ K'gX'^ decays is that 
the familon as a neutral very weakly-interacting particle escapes from the detector without 
any trace and only its light meson partner can be observed. Due to the two-body decay 
structure, the meson partner is produced with a well defined momentum of 2.65 GeV/c in 
the center of mass frame of the decaying B meson. However, in the lab frame its momentum 
is spread between 2.49 — 2.80 GeV/c due to Doppler broadening. Other detected particles 
and photons must be coming from the decay of the other B meson. Our analysis strategy 
to search for these decay modes is the following: (1) we select events with a well identified 
light meson having a momentum in the expected range while (2) all remaining particles must 
be consistent with the decay of a second B meson, and (3) eliminate as much continuum 
background as possible. 

Candidates for the vr^ or meson partner of the familon ("meson candidate") were 
selected from well- reconstructed tracks originating near the e^e~ interaction point (IP). Since 
charged vr and K meson separation in the momentum range expected is difficult with the 
CLEO detector, we combined the charged B decay modes by requiring the charged meson 
candidate's dE/dx to be consistent with either the pion or the kaon hypothesis within 2.5 
standard deviation (a). We rejected electrons based on dE/dx and the ratio of the track 
momentum to the associated shower energy deposited in the Csl calorimeter. Muons were 
rejected based on the penetration depth in the steel absorber surrounding the detector. The 
Kg candidates were reconstructed via their decay into tt+tt^ by requiring a decay vertex 



5 



displacement from the IP and an invariant tttt mass within 10 MeV/c^ of the known Kg 
mass. We accepted meson candidates with momentum in the range 2.49 < ph± < 2.81 
GeV/c or 2.47 < pj^o < 2.79 GeV/c. These and other selection criteria were optimized by 
maximizing the signal significance, S'^/{S + B), where S and B, the expected signal and 
background level was determined from Monte-Carlo simulated samples assuming a signal 
branching fraction of 10~^. 

Since the remaining particles in the event must originate from the decay of the second B 



meson we required that the beam constrained mass, M{B) = y-Ebeam (Z^Pi)^); be close 
to the B meson mass and the energy difference, AE = J2Ei— -Ebcam, be close to zero, where 
Ei and pi are the energy and momentum of all detected particles in the event except for the 
meson candidate. The optimization of the selection criteria on the M{B) and AE variables 
resulted in M{B) > 5.245 GeV/c^ (M(5) > 5.24 GeV/c^) and -2.1 < AE < 0.3 GeV 
(—3.0 < AE < 0.4 GeV) hmits for the charged (neutral) B decay mode. 

The main contribution to the background comes from continuum events. These events 
typically exhibit a two-jet structure and produce high momentum back-to-back tracks, while 
BB events tend to have a more isotropic decay structure, since the B mesons are produced 
nearly at rest (Pb ~ 0.32 GeV/c). We used the Fisher discriminant technique to reduce 
the continuum background. The Fisher discriminant was formed as the linear combination 
of 14 shape variables: 9 momentum flow variables (the sum of the momentum of all detected 
particles in 10° angular bins around the direction of the meson candidate); the angle between 
the momentum of the other B meson reconstructed from the rest of the event and the e^e^ 
collision ("beam") axis; the angle between the momentum of the meson candidate and the 
beam axis; the second order normalized Fox- Wolfram moment [|16|; the angle between the 
momentum of the meson candidate and the thrust axis of the rest of the event; and the 
maximum opening angle of the cone opposite to the momentum of the meson candidate, in 
which no other charged track, 7r° or Kg was detected. The combination coefficients were 
chosen to maximize the separation between the simulated signal and continuum background 
samples. 

The distribution of the Fisher discriminant used in the charged B analysis is shown 
for simulated events and off-resonance data on Fig. |^. The agreement between simulated 
continuum and off-resonance events is very good. We selected candidate events with a Fisher 
discriminant less than —0.4 in case of the charged B decay mode, and less than 1.0 for the 
neutral B decay mode. 

The overall signal selection efficiency is 7.2% for 5^ h^X^ and 6.6% for the B^ — >■ 
K^X^ events. The systematic error on the efficiency is 13% (18%) for the charged (neutral) 
B decay mode. The contributions to this error are due to the uncertainties in the tracking 
efficiency, 2% (4%), the momentum selection, 1% (1%), M{B) and AE selection, 6% (6%), 
Fisher discriminant restriction, 11% (16%), and limited Monte Carlo statistics, 1% (1%). 

Table | lists the number of events passing the consecutive selection requirements in the 
data and simulated signal samples. Figure ^ shows the momentum distribution of the meson 
candidate for on-resonance and off-resonance events along with the distributions for simu- 
lated events after all selection criteria except the tight momentum restriction on the meson 
candidate were applied. The number of on-resonance (off-resonance) events in the selected 
momentum range is 74 (32) in case of the B"^ h^X^ and 44 (14) in case of the B^ KgX^ 
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FIG. 1. Distribution of the Fisher discriminant used in the h^X^ analysis for simulated 

signal (solid) and continuum (dashed) as well as off-resonance data (points) samples. The his- 
tograms are normalized to the statistics of the off-resonance data. The signal histogram is plotted 
assuming a branching ratio of 50 x 10^^. The vertical line represents the optimum selection value 
below which events were accepted. 



analysis. The study of the background from 6 — > c, and other rare 6 — > u and b s decays 
as well as from tau decays using simulated data samples showed these to be negligible. 
We calculated the branching fraction based on 

where A^^on and A'ofr are the observed events in the signal region in the on-resonance and 
off-resonance data samples, respectively, R{= 2.0) is the normalization coefficient between 
the two samples, e is the signal selection efficiency, and Nb is the total number of charged 
(neutral) B mesons in the data sample, assuming equal production of charged and neutral 
B meson pairs from the T{AS) 0. We find B{B^ h^X^) = (1.4 ± 2.1) x 10"^ and 

TABLE I. Number of events passing each consecutive selection criteria in on-resonance, 
off-resonance data and simulated signal samples. 



On-res. Off-res. MC Signal On-res. Off-res. MC Signal 

Total events 57 million 23 million 180,000 57 milhon 23 million 90,000 

Pre-selected events 157,919 73,671 90,211 64,207 31,230 36,953 

Momentum selection 41,981 20,437 83,592 18,675 9,224 34,720 

Mb and AE selection 14,243 7,073 55,024 2,330 1,135 17,725 

Fisher selection 74 32 12,896 44 14 5,973 
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FIG. 2. Momentum distribution of the meson candidates. Filled and empty dots represent the 
on-resonance and the normalized off-resonance data, respectively. Solid histogram shows the pre- 
diction from e~^e~ qq plus b ^ c simulations while the dashed histogram shows the distribution 
from e^e" qq only. These histograms are normalized to the statistics of our data sample. Simu- 
lated signal events are shown by the dotted histogram assuming that B{B^ — > h^X^) 30 x 10~^ 
and B{B^ — > K^X^) r:^ 12 x 10~^. The accepted signal region is indicated by the arrows. 



B{B^ — > K^X^) = (2.5 ± 1.7) X 10~^. The error in the branching fraction is dominated by 
the statistical error in Non and Nos- We derived a 90% confidence level upper limit based 
on the frequentist approach applied for Gaussian data close to a physical boundary ||18|| : 
B{B^ h^X^) < 4.9 X 10-5 and B{B^ K%X^) < 5.3 x 10"^ 

The upper limits can be converted into a lower bound on the family symmetry breaking 
scale, = F/{g'^Ti,s(^j_^), with vector-like coupling between the familon and the quarks 

usmg Eq. II To do so we take the form factor -Fi(O) to be 0.25 from a sum rules calculation 
|19| |. The upper limit on the branching fraction of KgX^ gives F^^ ^ 6.4 x 10^ 

GeV. The other limit gives a slightly better bound of F^^^^^-^ > 1.3 x 10*^ GeV with the 
assumption that the familon couples to the d and s quark with approximately the same 
strength {Fbs ^ Fbd) ■ 

In conclusion, we performed a search for the decays B^ h^X^ and B^ K^X^, 
setting upper limits for the branching fractions at 4.9 x 10^^ and 5.3 x 10^^ respectively. 
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These limits constrain new physics leading to two-body B decays involving any massless 
neutral weakly-interacting particle X°. Applying the limit to the case where X° is a familon, 
we obtain the first lower bound on the family symmetry breaking scale involving the third 
generation of quarks at 10^ GeV. 

We gratefully acknowledge the effort of the CESR staff in providing us with excellent 
luminosity and running conditions. This work was supported by the National Science Foun- 
dation, the U.S. Department of Energy, the Research Corporation, the Natural Sciences and 
Engineering Research Council of Canada and the Texas Advanced Research Program. 
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